Reinforcement Learning with Replacing Eligibility

The eligibility trace is one of the basic mechanisms used in reinforcement learning to handle delayed reward. In this paper we introduce a new kind of eligibility trace, the replacing trace, analyze it theoretically, and show that it results in faster, more reliable learning than the conventional trace. Both kinds of trace assign credit to prior events according to how recently they occurred, but only the conventional trace gives greater credit to repeated events. Our analysis is for conventional and replace-trace versions of the ooine TD(1) algorithm applied to undiscounted absorbing Markov chains. First, we show that these methods converge under repeated presentations of the training set to the same predictions as two well known Monte Carlo methods. We then analyze the relative eeciency of the two Monte Carlo methods. We show that the method corresponding to conventional TD is biased, whereas the method corresponding to replace-trace TD is unbiased. In addition, we show that the method corresponding to replacing traces is closely related to the maximum likelihood solution for these tasks, and that its mean squared error is always lower in the long run. Computational results connrm these analyses and show that they are applicable more generally. In particular, we show that replacing traces sig-niicantly improve performance and reduce parameter sensitivity on the \Mountain-Car" task, a full reinforcement-learning problem with a continuous state space, when using a feature-based function approximator. Two fundamental mechanisms have been used in reinforcement learning to handle delayed reward. One is temporal-diierence (TD) learning, as in the TD() algorithm (Sutton, 1988) and in Q-learning (Watkins, 1989). TD learning in eeect constructs an internal reward signal that is less delayed than the original, external one. However, TD methods can eliminate the delay completely only on fully Markov problems, which are rare in practice. In most problems some delay always remains between an action and its eeective reward, and on all problems some delay is always present during the time before TD learning is complete. Thus, there is 2 a general need for a second mechanism to handle whatever delay is not eliminated by TD learning. The second mechanism that has been widely used for handling delay is the eligibility trace. 1 Introduced by Klopf (1972), eligibility traces have been used in a variety of reinforcement learning systems (e. Systematic empirical studies of eligibility traces in conjunction with TD methods were made by Sutton (1984), and theoretical …